<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">

<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
    
    <title>Building Networks with Modules and Connections &mdash; PyBrain v0.3 documentation</title>
    <link rel="stylesheet" href="../_static/default.css" type="text/css" />
    <link rel="stylesheet" href="../_static/pygments.css" type="text/css" />
    <script type="text/javascript">
      var DOCUMENTATION_OPTIONS = {
        URL_ROOT:    '../',
        VERSION:     '0.3',
        COLLAPSE_MODINDEX: false,
        FILE_SUFFIX: '.html',
        HAS_SOURCE:  true
      };
    </script>
    <script type="text/javascript" src="../_static/jquery.js"></script>
    <script type="text/javascript" src="../_static/doctools.js"></script>
    <link rel="top" title="PyBrain v0.3 documentation" href="../index.html" />
    <link rel="next" title="Classification with Feed-Forward Neural Networks" href="fnn.html" />
    <link rel="prev" title="Introduction" href="intro.html" /> 
  </head>
  <body>
    <div class="related">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../genindex.html" title="General Index"
             accesskey="I">index</a></li>
        <li class="right" >
          <a href="../modindex.html" title="Global Module Index"
             accesskey="M">modules</a> |</li>
        <li class="right" >
          <a href="fnn.html" title="Classification with Feed-Forward Neural Networks"
             accesskey="N">next</a> |</li>
        <li class="right" >
          <a href="intro.html" title="Introduction"
             accesskey="P">previous</a> |</li>
        <li><a href="../index.html">PyBrain v0.3 documentation</a> &raquo;</li> 
      </ul>
    </div>  

    <div class="document">
      <div class="documentwrapper">
        <div class="bodywrapper">
          <div class="body">
            
  <div class="section" id="building-networks-with-modules-and-connections">
<span id="netmodcon"></span><h1>Building Networks with Modules and Connections<a class="headerlink" href="#building-networks-with-modules-and-connections" title="Permalink to this headline">¶</a></h1>
<p>This chapter will guide you to use PyBrain&#8217;s most basic  structural elements:
the <tt class="xref docutils literal"><span class="pre">FeedForwardNetwork</span></tt> and <tt class="xref docutils literal"><span class="pre">RecurrentNetwork</span></tt> classes and with
them the <tt class="xref docutils literal"><span class="pre">Module</span></tt> class and the <tt class="xref docutils literal"><span class="pre">Connection</span></tt> class. We
have already seen how to create networks with the <tt class="docutils literal"><span class="pre">buildNetwork</span></tt> shortcut - but
since this technique is limited in some ways, we will now explore how to create
networks from the ground up.</p>
<div class="section" id="feed-forward-networks">
<h2>Feed Forward Networks<a class="headerlink" href="#feed-forward-networks" title="Permalink to this headline">¶</a></h2>
<p>We will start with a simple example, building a multi layer perceptron.</p>
<p>First we make a new <tt class="xref docutils literal"><span class="pre">FeedForwardNetwork</span></tt> object:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="k">from</span> <span class="nn">pybrain.structure</span> <span class="k">import</span> <span class="n">FeedForwardNetwork</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span> <span class="o">=</span> <span class="n">FeedForwardNetwork</span><span class="p">()</span>
</pre></div>
</div>
<p>Next, we&#8217;re constructing the input, hidden and output layers:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="k">from</span> <span class="nn">pybrain.structure</span> <span class="k">import</span> <span class="n">LinearLayer</span><span class="p">,</span> <span class="n">SigmoidLayer</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">inLayer</span> <span class="o">=</span> <span class="n">LinearLayer</span><span class="p">(</span><span class="mf">2</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">hiddenLayer</span> <span class="o">=</span> <span class="n">SigmoidLayer</span><span class="p">(</span><span class="mf">3</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">outLayer</span> <span class="o">=</span> <span class="n">LinearLayer</span><span class="p">(</span><span class="mf">1</span><span class="p">)</span>
</pre></div>
</div>
<p>There are a couple of different classes of layers. For a complete list check out
the <tt class="xref docutils literal"><span class="pre">modules</span></tt> package.</p>
<p>In order to use them, we have to add them to the network:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">addInputModule</span><span class="p">(</span><span class="n">inLayer</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">addModule</span><span class="p">(</span><span class="n">hiddenLayer</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">addOutputModule</span><span class="p">(</span><span class="n">outLayer</span><span class="p">)</span>
</pre></div>
</div>
<p>We can actually add multiple input and output modules. The net has to know
which of its modules are input and output modules, in order to forward propagate
input and to back propagate errors.</p>
<p>It still needs to be explicitly determined how they should be connected. For
this we use the most common connection type, which produces a full connectivity
between layers, by connecting each neuron of one layer with each neuron of the
other. This is implemented by the <tt class="xref docutils literal"><span class="pre">FullConnection</span></tt> class:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="k">from</span> <span class="nn">pybrain.structure</span> <span class="k">import</span> <span class="n">FullConnection</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">in_to_hidden</span> <span class="o">=</span> <span class="n">FullConnection</span><span class="p">(</span><span class="n">inLayer</span><span class="p">,</span> <span class="n">hiddenLayer</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">hidden_to_out</span> <span class="o">=</span> <span class="n">FullConnection</span><span class="p">(</span><span class="n">hiddenLayer</span><span class="p">,</span> <span class="n">outLayer</span><span class="p">)</span>
</pre></div>
</div>
<p>As with modules, we have to explicitly add them to the network:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">addConnection</span><span class="p">(</span><span class="n">in_to_hidden</span><span class="p">)</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">addConnection</span><span class="p">(</span><span class="n">hidden_to_out</span><span class="p">)</span>
</pre></div>
</div>
<p>All the elements are in place now, so we can do the final step that makes our
MLP usable,     which is to call the <tt class="docutils literal"><span class="pre">.sortModules()</span></tt> method:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">sortModules</span><span class="p">()</span>
</pre></div>
</div>
<p>This call does some internal initialization which is necessary before the net
can finally be used: for example, the modules are sorted topologically.</p>
</div>
<div class="section" id="examining-a-network">
<h2>Examining a Network<a class="headerlink" href="#examining-a-network" title="Permalink to this headline">¶</a></h2>
<p>We can actually print networks and examine their structure:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="k">print</span> <span class="n">n</span>
<span class="go">FeedForwardNetwork-6</span>
<span class="go">Modules:</span>
<span class="go"> [&lt;LinearLayer &#39;LinearLayer-3&#39;&gt;, &lt;SigmoidLayer &#39;SigmoidLayer-7&#39;&gt;, &lt;LinearLayer &#39;LinearLayer-8&#39;&gt;]</span>
<span class="go">Connections:</span>
<span class="go"> [&lt;FullConnection &#39;FullConnection-4&#39;: &#39;LinearLayer-3&#39; -&gt; &#39;SigmoidLayer-7&#39;&gt;, &lt;FullConnection &#39;FullConnection-5&#39;: &#39;SigmoidLayer-7&#39; -&gt; &#39;LinearLayer-8&#39;&gt;]</span>
</pre></div>
</div>
<p>Note that the output on your machine will not necessarily be the same.</p>
<p>One way of using the network is to call its &#8216;activate()&#8217; method with an input to
be transformed:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">activate</span><span class="p">([</span><span class="mf">1</span><span class="p">,</span> <span class="mf">2</span><span class="p">])</span>
<span class="go">array([-0.11302355])</span>
</pre></div>
</div>
<p>Again, this might look different on your machine - the weights of the
connections have already been initialized randomly. To have a look at those
parameters, just check the <tt class="docutils literal"><span class="pre">.params</span></tt> field of the connections:</p>
<p>We can access the trainable parameters (weights) of a connection directly, or
read all weights of the network at once:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">in_to_hidden</span><span class="o">.</span><span class="n">params</span>
<span class="go">array([ 1.37751406,  1.39320901, -0.24052686, -0.67970042, -0.5999425 , -1.27774679])</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">hidden_to_out</span><span class="o">.</span><span class="n">params</span>
<span class="go">array([-0.32156782,  1.09338421,  0.48784924])</span>
</pre></div>
</div>
<p>The network encapsulating the modules actually holds the parameters too. You can
check them out here:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">params</span>
<span class="go">array([ 1.37751406,  1.39320901, -0.24052686, -0.67970042, -0.5999425 ,</span>
<span class="go">     -1.27774679, -0.32156782,  1.09338421,  0.48784924])</span>
</pre></div>
</div>
<p>As you can see, the last three parameters of the network equal the parameters of
the second connection.</p>
</div>
<div class="section" id="naming-your-networks-structure">
<h2>Naming your Networks structure<a class="headerlink" href="#naming-your-networks-structure" title="Permalink to this headline">¶</a></h2>
<p>In some settings it makes sense to give the parts of a network explicit
identifiers. The structural components are derive from the <tt class="xref docutils literal"><span class="pre">Named</span></tt>
class, which means that they have an attribute <cite>.name</cite> by which you can
identify it by. If no name is given, a new name will be generated automatically.</p>
<p>Subclasses can also be named by passing the <cite>name</cite> argument on initialization:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">LinearLayer</span><span class="p">(</span><span class="mf">2</span><span class="p">)</span>
<span class="go">&lt;LinearLayer &#39;LinearLayer-11&#39;&gt;</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">LinearLayer</span><span class="p">(</span><span class="mf">2</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">&quot;foo&quot;</span><span class="p">)</span>
<span class="go">&lt;LinearLayer &#39;foo&#39;&gt;</span>
</pre></div>
</div>
<p>By using names for your networks, printouts look more concise and readable. They
also ensure that your network components are named in the same way every time
you run your program.</p>
</div>
<div class="section" id="using-recurrent-networks">
<h2>Using Recurrent Networks<a class="headerlink" href="#using-recurrent-networks" title="Permalink to this headline">¶</a></h2>
<p>In order to allow recurrency, networks have to be able to &#8220;look back in time&#8221;.
Due to this, the <tt class="xref docutils literal"><span class="pre">RecurrentNetwork</span></tt> class is different from the
<tt class="xref docutils literal"><span class="pre">FeedForwardNetwork</span></tt> class in the substantial way, that the complete
history is saved. This is actually memory consuming, but necessary for some
learning algorithms.</p>
<p>To create a recurrent network, just do as with feedforward networks but use the
appropriate class:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="k">from</span> <span class="nn">pybrain.structure</span> <span class="k">import</span> <span class="n">RecurrentNetwork</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span> <span class="o">=</span> <span class="n">RecurrentNetwork</span><span class="p">()</span>
</pre></div>
</div>
<p>We will quickly build up a network that is the same as in the example above:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">addInputModule</span><span class="p">(</span><span class="n">LinearLayer</span><span class="p">(</span><span class="mf">2</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">&#39;in&#39;</span><span class="p">))</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">addModule</span><span class="p">(</span><span class="n">SigmoidLayer</span><span class="p">(</span><span class="mf">3</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">&#39;hidden&#39;</span><span class="p">))</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">addOutputModule</span><span class="p">(</span><span class="n">LinearLayer</span><span class="p">(</span><span class="mf">1</span><span class="p">,</span> <span class="n">name</span><span class="o">=</span><span class="s">&#39;out&#39;</span><span class="p">))</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">addConnection</span><span class="p">(</span><span class="n">FullConnection</span><span class="p">(</span><span class="n">n</span><span class="p">[</span><span class="s">&#39;in&#39;</span><span class="p">],</span> <span class="n">n</span><span class="p">[</span><span class="s">&#39;hidden&#39;</span><span class="p">],</span> <span class="n">name</span><span class="o">=</span><span class="s">&#39;c1&#39;</span><span class="p">))</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">addConnection</span><span class="p">(</span><span class="n">FullConnection</span><span class="p">(</span><span class="n">n</span><span class="p">[</span><span class="s">&#39;hidden&#39;</span><span class="p">],</span> <span class="n">n</span><span class="p">[</span><span class="s">&#39;out&#39;</span><span class="p">],</span> <span class="n">name</span><span class="o">=</span><span class="s">&#39;c2&#39;</span><span class="p">))</span>
</pre></div>
</div>
<p>The <tt class="xref docutils literal"><span class="pre">RecurrentNetwork</span></tt> class has one additional method,
<tt class="docutils literal"><span class="pre">.addRecurrentConnection()</span></tt>, which looks back in time one timestep. We can
add one from the hidden to the hidden layer:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">addRecurrentConnection</span><span class="p">(</span><span class="n">FullConnection</span><span class="p">(</span><span class="n">n</span><span class="p">[</span><span class="s">&#39;hidden&#39;</span><span class="p">],</span> <span class="n">n</span><span class="p">[</span><span class="s">&#39;hidden&#39;</span><span class="p">],</span> <span class="n">name</span><span class="o">=</span><span class="s">&#39;c3&#39;</span><span class="p">))</span>
</pre></div>
</div>
<p>If we now activate the network, we will get different outputs each time:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">sortModules</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">activate</span><span class="p">((</span><span class="mf">2</span><span class="p">,</span> <span class="mf">2</span><span class="p">))</span>
<span class="go">array([-0.1959887])</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">activate</span><span class="p">((</span><span class="mf">2</span><span class="p">,</span> <span class="mf">2</span><span class="p">))</span>
<span class="go">array([-0.19623716])</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">activate</span><span class="p">((</span><span class="mf">2</span><span class="p">,</span> <span class="mf">2</span><span class="p">))</span>
<span class="go">array([-0.19675801])</span>
</pre></div>
</div>
<p>Of course, we can clear the history of the network. This can be done by calling
the <cite>reset</cite> method:</p>
<div class="highlight-python"><div class="highlight"><pre><span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">reset</span><span class="p">()</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">activate</span><span class="p">((</span><span class="mf">2</span><span class="p">,</span> <span class="mf">2</span><span class="p">))</span>
<span class="go">array([-0.1959887])</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">activate</span><span class="p">((</span><span class="mf">2</span><span class="p">,</span> <span class="mf">2</span><span class="p">))</span>
<span class="go">array([-0.19623716])</span>
<span class="gp">&gt;&gt;&gt; </span><span class="n">n</span><span class="o">.</span><span class="n">activate</span><span class="p">((</span><span class="mf">2</span><span class="p">,</span> <span class="mf">2</span><span class="p">))</span>
<span class="go">array([-0.19675801])</span>
</pre></div>
</div>
<p>After the call to <tt class="docutils literal"><span class="pre">.reset()</span></tt>, we are getting the same outputs as just after
the objects creation.</p>
</div>
</div>


          </div>
        </div>
      </div>
      <div class="sphinxsidebar">
        <div class="sphinxsidebarwrapper">
            <p class="logo"><a href="../index.html">
              <img class="logo" src="../_static/pybrain_logo.gif" alt="Logo"/>
            </a></p>
            <h3><a href="../index.html">Table Of Contents</a></h3>
            <ul>
<li><a class="reference external" href="">Building Networks with Modules and Connections</a><ul>
<li><a class="reference external" href="#feed-forward-networks">Feed Forward Networks</a></li>
<li><a class="reference external" href="#examining-a-network">Examining a Network</a></li>
<li><a class="reference external" href="#naming-your-networks-structure">Naming your Networks structure</a></li>
<li><a class="reference external" href="#using-recurrent-networks">Using Recurrent Networks</a></li>
</ul>
</li>
</ul>

            <h4>Previous topic</h4>
            <p class="topless"><a href="intro.html"
                                  title="previous chapter">Introduction</a></p>
            <h4>Next topic</h4>
            <p class="topless"><a href="fnn.html"
                                  title="next chapter">Classification with Feed-Forward Neural Networks</a></p>
            <h3>This Page</h3>
            <ul class="this-page-menu">
              <li><a href="../_sources/tutorial/netmodcon.txt"
                     rel="nofollow">Show Source</a></li>
            </ul>
          <div id="searchbox" style="display: none">
            <h3>Quick search</h3>
              <form class="search" action="../search.html" method="get">
                <input type="text" name="q" size="18" />
                <input type="submit" value="Go" />
                <input type="hidden" name="check_keywords" value="yes" />
                <input type="hidden" name="area" value="default" />
              </form>
              <p class="searchtip" style="font-size: 90%">
              Enter search terms or a module, class or function name.
              </p>
          </div>
          <script type="text/javascript">$('#searchbox').show(0);</script>
        </div>
      </div>
      <div class="clearer"></div>
    </div>
    <div class="related">
      <h3>Navigation</h3>
      <ul>
        <li class="right" style="margin-right: 10px">
          <a href="../genindex.html" title="General Index"
             >index</a></li>
        <li class="right" >
          <a href="../modindex.html" title="Global Module Index"
             >modules</a> |</li>
        <li class="right" >
          <a href="fnn.html" title="Classification with Feed-Forward Neural Networks"
             >next</a> |</li>
        <li class="right" >
          <a href="intro.html" title="Introduction"
             >previous</a> |</li>
        <li><a href="../index.html">PyBrain v0.3 documentation</a> &raquo;</li> 
      </ul>
    </div>
    <div class="footer">
      &copy; Copyright 2009, CogBotLab &amp; Idsia.
      Last updated on Nov 12, 2009.
      Created using <a href="http://sphinx.pocoo.org/">Sphinx</a> 0.6.3.
    </div>
  </body>
</html>